1. Field of the Invention
The present invention relates to processing to obtain arrangement information for each of markers arranged in a three-dimensional space.
2. Description of the Related Art
Recently, research of Mixed Realty (MR) technology has been developed. The MR technology is intended to merge a physical space and a virtual space, which is created by a computer, with each other without including seams. Of the MR technology, Augmented Reality (AR) technology for displaying the virtual space in a superimposed relation to the physical space has particularly received attention.
There are two types of image display apparatus for use in the AR technology, i.e., the video see-through type and the optical see-through type. In the image display apparatus of the video see-through type, a combined image is displayed by drawing, on an image of a physical space captured by an image pickup apparatus, an image of a virtual space produced depending on the position and the posture of the image pickup apparatus (such as character information and a virtual object drawn using computer graphics) in a superimposed manner. In the image display apparatus of the optical see-through type, an image of a virtual space produced depending on the position and the posture of an observer's viewpoint is displayed on an optical see-through display attached to the observer's head.
The AR technology is expected to be usefully employed in various fields including a surgical aid to display a situation inside the body of a patient on the body surface of the patient in a superimposed manner, simulation of building construction to display a virtual building on vacant land in a superimposed manner, and an assembly aid to display operation procedures and layouts of wiring in assembly work.
One of the most important problems to be overcome in the AR technology is how precisely the physical space and the virtual space are registered with each other, and a variety of approaches have been proposed so far. The problem of registration in the AR technology is equivalent to, in the video see-through type, a problem of precisely determining the position and the posture of the image pickup apparatus in a scene (i.e., in a reference coordinate system defined in the scene). Also, the problem of registration in the AR technology is equivalent to, in the optical see-through type, a problem of precisely determining the viewpoint of the observer or the position and the posture of the display in a scene.
As one method for overcoming the problem of registration in the AR technology of the video see-through type, it is generally performed to determine the position and the posture of a measurement target by using a plurality of markers arranged in a scene. More specifically, the position and the posture of the image pickup apparatus in the reference coordinate system are determined based on the correspondent relationship between a projected position of each marker in a captured image and a marker position in the reference coordinate system, the latter being known information.
Also, according to one method for overcoming the problem in the AR technology of the optical see-through type, the image pickup apparatus is attached to the measurement target (i.e., the observer's head or the display). Further, the position and the posture of the image pickup apparatus are determined in a similar manner to that in the video see-through type.
In the field of photo survey, the method of determining the position and the posture of the image pickup apparatus based on the correspondent relationship between the projected image of the marker on the captured image and the three-dimensional position of the marker is described, for example, in R. M. Haralick, C. Lee, K. Ottenberg, and M. Nolle: “Review and Analysis of Solutions of the Three Point Perspective Pose Estimation Problem”, International Journal of Computer Vision, vol. 13, No. 3, PP. 331-356, 1994. It is also proposed to optimize the position and the posture of the image pickup apparatus by using, as an initial value, the position and the posture of the image pickup apparatus which are determined based on the projected image of the marker on the captured image as described above. The optimization is performed by repetitive calculations executed so as to minimize an error between the actually observed position of the projected image of the marker on the captured image and the calculated position of the projected image, i.e., an error between the three-dimensional position of the marker and the position of the projected image calculated from the position and the posture of the image pickup apparatus. Such a method is described, for example, in H. Kato, M. Billinghurst, K. Asano, and K. Tachibana: “An Augmented Reality System and its Calibration based on Marker Tracking”, Transactions of Virtual Reality Society of Japan, Vol. 4, No. 4, PP. 607-616, 1999. Thus, by using the above-mentioned methods, the position and the posture of the image pickup apparatus have been determined so far based on the image captured by the image pickup apparatus.
On the other hand, as described in, e.g., Japanese Patent Laid-Open No. 11-084307, Japanese Patent Laid-Open No. 2000-041173, and A. State, G. Hirota, D. T. Chen, W. F. Garrett and M. A. Livingston: “Superior Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking”, Proc. SIGGRAPH '96, PP. 429-438, 1996, there is known a method of attaching a position and posture sensor having 6 degrees of freedom, e.g., a magnetic sensor or an ultrasonic sensor, to the image pickup apparatus which is a measurement target, and measuring the position and the posture of the image pickup apparatus in combination with the above-described detection of the marker based on image processing. Because an output value of the sensor can be stably obtained in spite of accuracy being changed depending on a measurement range, the method using the sensor and the image processing in a combined manner is more robust than the method using only the image processing. In Japanese Patent Laid-Open No. 2000-041173, the position and the posture of the image pickup apparatus are obtained by using, as an initial value, the position and the posture of the image pickup apparatus obtained from the position and posture sensor having 6 degrees of freedom, and by minimizing the error between the observed position and the calculated position of the marker on the projected image of the marker on the captured image through repetitive calculations.
The above-described registration method using the marker requires the following information in order to determine the position and the posture of the image pickup apparatus, as the measurement target, in the reference coordinate system. For example, in the case of a point-like marker (hereinafter referred to as a “point marker”) in the form of a circular area located in a space and filled in with a single color, the position of the center of gravity of the point marker in the reference coordinate system and the posture of the marker with respect to the reference coordinate system are required to be known. In the case of the marker having a polygonal shape, e.g., a triangular or square shape, the position of the center of gravity of the marker in the reference coordinate system and the posture of the marker with respect to the reference coordinate system are required to be known. When the polygonal marker, e.g., a square marker, is employed, the marker is itself used as a reference for a coordinate system without separately setting the reference coordinate system in many cases. However, when a plurality of polygonal markers are used, the reference coordinate system is required because the relative relationships among the positions and the postures of those polygonal markers are required to be known.
Although the position and the posture of the marker can be measured by hand work using a tape measure and a protractor or a measuring instrument, the measurement is usually performed by utilizing an image in consideration of that the hand work accompanies the problem of accuracy and the need of time and effort. The position of the point marker can be measured by the so-called bundle adjustment method. According to the bundle adjustment method, the position of the point marker is measured through the following processing. Many images of the point marker are captured by an image pickup apparatus. The position of the point marker and the position and the posture of the image pickup apparatus are repeatedly calibrated so that an error between the observed position of a projected image of the point marker on each captured image and the calculated position of the projected image of the point marker (i.e., a projection error) is minimized, the latter being calculated based on the three-dimensional position of the point marker and the position and the posture of the image pickup apparatus.
Further, G. Baratoff, A. Neubeck and H. Regenbrecht: “Interactive Multi-Marker Calibration for Augmented Reality Applications”, Proc. ISMAR 2002, PP. 107-116, 2002 (hereinafter “Baratoff et al.”) describes a method of measuring the positions and the postures of many square markers arranged in a three-dimensional space. According to the method described in Baratoff et al, images of the many square markers arranged in the three-dimensional space are captured in large number. The position and the posture of the image pickup apparatus used to capture each image and the positions and the postures of the square markers are determined through repetitive calculations so that a projection error of each square marker image is minimized.
When the method of measuring the position of the point marker is performed using the above-described bundle adjustment method, the accurate position and posture cannot be determined unless at least two images are captured for each marker. In the case of the point marker, if only one image of the point marker is captured, a linear line containing the point marker in the three-dimensional space is decided and the position of the image pickup apparatus in the direction of depth looking from an observer is not definite. Further, when the measurement of the position and the posture is performed for the square marker, the following tendency is also resulted similarly to the case of the point marker. As the number of the captured images increases, an error is distributed to a larger number of images, and the position and the posture of the marker can be measured with higher accuracy than when a smaller number of images are captured. It is therefore desired that the image of the marker for which the position and the posture are to be determined is captured as many times as possible.
In the known marker measurement, however, there is no method of enabling an observer to confirm information regarding how many times each marker image has been captured up to that time while observing the image captured by the image pickup apparatus.
The position and the posture of the marker can be measured with higher accuracy by, in addition to increasing the number of captured images, observing the marker from directions in larger number instead of one direction.
In the known marker measurement, however, there is no method of enabling an observer to confirm information regarding in which directions each marker image has been captured up to that time while observing the image captured by the image pickup apparatus.